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[Title of Document] Specification 

[Title of the Invention] Controller for Photographing 
Apparatus and Photographing System 
[Scope of Claims for a Patent] 
[Claim 1] 

A controller for a photographing apparatus having 
a photographing portion with driving means that allows the 
photographing direction of photographing means to be varied, 
comprising : 

displayingmeans for displaying a panoramapicture 
generated with a picture photographed by the photographing 
means; and 

controlling means for referencing the panorama 
picture and varying the photographing direction of the 
photographing means. 
[Claim 2] 

A controller for a photographing apparatus having 
a photographing portion with driving means that allows the 
photographing direction of photographing means to be varied, 
the controller comprising: 

an operation area in which a panorama picture 
generated with a picture photographed by the photographing 
means is displayed; and 

picture selecting means for allowing the user to 
designate a desired point in said operation area, selecting 
an object photographed by the photographing means 
correspondingtothedesignatedpoint , andmovingtheselected 



obj ect t odes iredpositional coordinates of the drivingmeans . 
[Claim 3] 

A controller for a photographing apparatus having 
a photographing portion with driving means that allows the 
photographing direction of photographing means to be varied, 
the controller comprising: 

an operation area in which a panorama picture 
generated with a picture photographed by the photographing 
means is displayed; and 

picture selecting means for allowing the user to 
designate a desired area in said operation area, selecting 
an object photographed by the photographing means 
corresponding to the designated area, and moving an object 
at the position corresponding to a desired point generated 
with the desired area to desired positional coordinates of 
the driving means. 
[Claim 4] 

The controller as set forth in claim 1, 2, or 3 , 
wherein the photographing portion is connected 
through a transmission line. 
[Claim 5] 

The controller as set forth in claim 1, 2, or 3 , 
wherein the current position of the driving means 
is displayed on the panorama picture. 
[Claim 6] 

The controller as set forth in claim 1, 2, or 3 , 
wherein the moving range of the driving means is 



displayed on the panorama picture. 
[Claim 7] 

The controller as set forth in claim 1, 2, or 3, 
wherein the angle of view of the photographing 
means is displayed on the panorama picture. 
[Claim 8] 

The controller as set forth in claim 1, 2, or 3, 
wherein said picture selecting means has a second 

operation area in which a current picture photographed by 

the photographing means is displayed, and 

wherein an object at a designatedposit ion is moved 

to desired positional coordinates of the driving means in 

the second operation area. 
[Claim 9] 

The controller as set forth in claim 1, 2, or 3, 
further comprising : 

commanding means for commanding the generation 
of the panorama picture. 
[Claim 10] 

A photographing system having a photographing 
portion with driving means that allows the photographing 
directionof photographingmeans tobe variedanda controller 
for a photographing apparatus, the controller controlling 
the photographing portion, 

wherein the controller comprises: 
an operation area in which a panorama picture 
generated with a picture photographed by the photographing 



means is di splayed ; and 

picture selecting means for selecting an object 
photographed by the photographing means in said operation 
area and moving the selected object to desired positional 
coordinates of the driving means . 
[Claim 11] 

The photographing system as set forth in claim 

10, 

wherein the photographing portion and the 
photographingcontrollerareconnectedthroughatransmission 
line . 

[Claim 12] 

The photographing system as set forth in claim 
10 , further comprising : 

commanding means for commanding the generation 
of the panorama picture . 
[Detailed Description of the Invention] 
[0001] 

[Technical Field to which the Invention belongs] 

The present invention relates to a controller for 
a photographing apparatus and a photographing system with 
a high operational characteristic and a high visibility 
suitable for a photographing operation of the apparatus that 
is disposed at a remote place and that is used for a monitoring 
operation, an observing operation, a guiding operation, a 
presenting operation, and so forth. 
[0002] 



[Prior Art] 

As shown in Fig. 23, when the user controls a 
photographing apparatus disposed at a remote place, he or 
she operates a pan tilter in eight directions (up, down, 
left, right, upper right, lower right, upper left, and lower 
left directions) with eight-direction keys, a zooming 
controller, and a wide-angle controller so as to photograph 
a desired object while observing a photographed picture 6A 
on a monitor 2. In the structure shown in Fig. 23, the user 
moves a cursor 7 to one of the direction keys 10 with a mouse 
8 and clicks the mouse . Thereby, the user drives a pan tilter 
to the designated direction. Alternatively, after the user 
has controlledaphotographing apparatus disposedat a remote 
place in the above-describedmethodandregisteredpan tilter 
information and zoom information of positions of pictures 
tobephotographed, heorshedrivesthephotographingapparatus 
at absolute positions corresponding to the registered 
positions so as to select pictures. 
[0003] 

[Subject that the Invention is to solve] 

In the conventional controller, a picture that 
is displayed on the monitor is limited in the range of which 
the photographing apparatus i s moved by the pan tilter. Thus , 
when the user photographs a desired object, he or she should 
operatethepantilterinthe full range thereof . Consequently, 
the user should have skill in operating the pan tilter. 
[0004] 
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When the user changes the photographing direction 
with the conventional direction keys , even if he or she stops 
pressing the direction keys, since the pan tilter does not 
immediately stops and thereby he or she may not catch a desired 
object . Whenthedirectionvaryingspeedof thephotographing 
apparatus with the pan tilter is low, although such a problem 
maybesolved, sincetheresponsecharacteristicdeteriorates, 
a high operational characteristic cannot be obtained. 
[0005] 

When the user wants to place a desired object at 
the center of the angl e of view of the photographing apparatus , 
since he or she controls the photographing direction while 
observing a picture on the monitor , he or she should determine 
the photographing direction on trial and error basis . Thus, 
theusermayspendalongtime for control ling thephotographing 
apparatus . Moreover , to properly operate the photographing 
apparatus, the user should have skill. 
[0006] 

When picture and control information is exchanged 
with a photographing apparatus disposed at a remote place 
through a low-capacity network, the control information may 
be lost and/or picture information may be delayed due to 
an irregularity of their arrival intervals. If the 
displacement of the pan tilter or the zooming controller 
is operated for picture and control information that has 
been delayed or lost, even if the user causes the pan tilter 
and the zooming controller to place the ob j ect at the desired 



position, the pan tilter and the zooming controller do not 
properly operate . Thus , the ob j ect is placed at an improper 
position due to the delay. In addition, depending on the 
line condition, the arrival intervals of picture information 
vary. Thus, the user should control the pan tilter and the 
zooming controller based on a prediction. Consequently, 
the user cannot properly control the pan tilter and the zooming 
controller . 
[0007] 

Therefore, an object of the present invention is 
to provide a controller for a photographing apparatus for 
allowing the user to designate a desiredposition or a desired 
area on a panorama picture displayed as a part or all the 
moving range of a pan tilter so that the user can easily 
obtain a desired picture with the photographing apparatus . 
[0008] 

Anotherobj ect of thepresent invent ionistoprovide 
a controller for aphotographing apparatus andaphotographing 
system with a high visibility and a high operational 
characteristic that allow the user to designate a desired 
position or a desired area on a screen and select an object 
with the designated position or area and the photographing 
apparatus to place the selected object at the center of the 
screen . 

[0009] 

[Means for Solving the Subject] 

The present invention described in claim 1 is a 
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controller for a photographing apparatus having a 
photographing portion with driving means that allows the 
photographing direction of photographing means to be varied , 
comprisingadisplayingmeansf or display ingapanoramapicture 
generated with a picture photographed by the photographing 
means, and a controlling means for referencing the panorama 
picture and varying the photographing direction of the 
photographing means . 
[0010] 

The present invention described in claim 2 is a 
controller for a photographing apparatus having a 
photographing portion with driving means that allows the 
photographing direction of photographing means to be varied, 
the control ler comprisinganoperationareainwhichapanorama 
picture generated with a picture photographed by the 
photographing means is displayed, and a picture selecting 
means for allowing the user to designate a desired point 
in the operation area, selecting an object photographed by 
thephotographingmeanscorrespondingtothedesignatedpoint , 
and moving the selected object to desired positional 
coordinates of the driving means . 
[0011] 

The present invention described in claim 3 is a 
controller for a photographing apparatus having a 
photographing portion with driving means that allows the 
photographing direct ion of photographing means to be varied, 
the control ler comprising an operationareainwhichapanorama 
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picture generated with a picture photographed by the 
photographing means is displayed, and a picture selecting 
means for allowing the user to designate a desired area in 
the operation area, selecting an object photographed by the 
photographing means corresponding to the designated area, 
and moving an object at theposition corresponding toadesired 
point generated with the desired area to desired positional 
coordinates of the driving means. 
[0012] 

The present invention describe in claim 10 is a 
photographing system having a photographing portion with 
driving means that allows the photographing direction of 
photographing means to be varied and a controller for a 
photographing apparatus, the controller controlling the 
photographing portion, wherein the controller comprises an 
operation area in which a panorama picture generated with 
apicturephotographedbythephotographingmeansisdi splayed, 
and a picture selecting means for selecting an object 
photographed by the photographing means in the operation 
area and moving the selected object to desired positional 
coordinates of the driving means . 
[0013] 

A picture photographed by a pan tilter camera that 
is disposed at a remote place and that can be moved in various 
directions is sent to a computer. The picture is displayed 
as a panorama picture in a display area of a monitor. The 
direction of a picture selecting means corresponding to the 
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direction of an object to be placed at the center of the 
angle of view of the photographing apparatus in the panorama 
picture is designated by a pointing device connected to the 
computer . Since the pan tilter is controlled with reference 
to thepanoramapicture, adesiredpicture canbe photographed 
by the photographing apparatus. 
[0014] 

In addition, the environment of the place at which 
the pan tilter camera is disposed is displayed as a panorama 
picture in the panorama operation area of the monitor of 
the computer. A desired point to be placed at the center 
of the angle of view of the photographing apparatus in a 
picture of the panorama operation area or a desired point 
generated with a desired area is designated by the pointing 
device connected to the computer. Thus, in the method of 
which the result is input, a selected object can be easily 
placed at the center of the screen. In addition, since a 
desiredpoint in the operation area on the screen or a desired 
point generated with a desired area is designated with the 
point ingdevice, theuser caneasilyknowthedrivingdirection 
of thepantilter camera . Inaddit ion tothepanorama operation 
area, another operation area for a picture maybe displayed. 
[0015] 

[Embodiment of the Invention] 

Next , with reference to the accompanying drawings , 
embodiments of the present invention will be described . Fig . 
1 shows an outline of the structure of a system according 
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to a first embodiment of the present invention. A monitor 
2 and a mouse 8 are connected to a computer 1 . The computer 
1 controls the driving operation of a pan tiler camera 3 
disposed at a remote place. In other words, a controller 
for the photographing apparatus is composed of the computer 
1. 

[0016] 

The pan tilter camera 3 is integrally composed 
of a pan tilter portion and a camera portion. In Fig. 1, 
the pan tilter camera 3 is disposed on a real scene as denoted 
by 4 . A screen of a picture photographed by the pan tilter 
camera 3 is denotedby 5 . This screen is hereinafter referred 
to as a photographed screen. The photographed screen 5 is 
an actually photographed screen. When a zoom lens of the 
pan tilter camera 3 is placed on the telephotograph side, 
the angle of view decreases. In contrast, when the zoom 
lens of the pan tilter camera 3 is placed on the wide-angle 
side, the angle of view increases. 
[0017] 

A picture on a photographed screen 5 captured by 
a pan tilter camera 3 is sent to a computer 1 through a video 
cable or the like. The picture data sent to the computer 
1 is decoded and displayed on a monitor 2. The monitor 2 
displays the photographed screen 5 in an operation area 6A 
on the monitor 2 . A panorama picture including the picture 
photographed by the pan tilter camera 3 is displayed in a 
panorama operation area 6B . An arrow-shaped cursor 7 is 
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displayed at the position of a mouse pointer of a mouse 8 
in the operation area 6A or the panorama display area 6B . 
The user designates a desired point or a desired area in 
the operation area 6A or the panorama operation area 6B with 
the mouse 8 so as to operate the pan tilter camera 3 . In 
the panorama operation area 6B, a frame 6C that represents 
the current position and the angle of view of the pan tilter 
and a pan tilter limiter 6D are superimposed to the panorama 
picture. The pan tilter limiter 6D represents the moving 
range of the pan tilter camera . In addition, when necessary , 
a panorama generation button 6E is displayed on the monitor 
2 . 

[0018] 

As shown in Fig. 2, the operation area 6A and the 
panorama operation area 6B are displayed on the monitor 2. 
With the mouse 8 , the user can move the cursor 7 and designate 
a desired point or a desired point generated with a desired 
area in the operation area 6A or the panorama operation area 
6B. The user operates the pan tiler so that an object 
corresponding to the designated point is placed at the center 
of the operation area 6A. In other words, when the user 
inputs a result to be displayed, an object selected 
corresponding to the input data is displayed at the center 
of the operation area 6A. 
[0019] 

Fig. 3 is ablockdiagramshowing the overall system 
according to the embodiment of the present invention. The 
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system shown in Fig. 3 comprises a camera portion 11, a pan 
tilter portion 12, a TV monitor 13 , a computer 1, a pointing 
device 14 (such as a mouse 8), and a monitor 2. The pan 
tilter camera 3 comprises a camera portion 11 and a pan tilter 
portion 12 . For example, the camera portion 11 is disposed 
on the pan tilter portion 12 . The camera port ion 11 comprises 
a lens block portion 15, a zoom lens 16, a zoom portion 17, 
a zoom lens motor 18, a solid state image pickup device 19, 
asignalseparating/automaticgainadjustingcircuit (SH/AGC) 
20, an A/D converter 21, and a signal processing circuit 

22. The camera portion 11 represents a video camera. 

[0020] 

Thepantilterportionl2comprisesamodecont roller 

23, a camera controller 24, a pan tilter controller 25, a 
pan motor 26, a tilt motor 27, and a pan tilter 28. The 
computer 1 comprises a control ling port ion 3 1 , a video capture 
portion 29, and a storing portion 30. The video capture 
portion 29 is composed of a video capture board. 

[0021] 

Rays emitted from an object are focused to the 
solid state image pickup device 19 through a lens set and 
a diaphragm of the lens block portion 15. An example of 
the sol id state image pickup device 19 is a CCD (Charge Coupled 
Device) . The focused rays (field picture) are converted 
into a picture signal and then sent to the signal 
separating/ automatic gain adjusting circuit 20 . The signal 
separating/automaticgainadjustingcircuit2 0samples/holds 
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the picture signal and controls the gain of the picture signal 
with a control signal of an auto iris (AE) . The resultant 
picture signal is sent to the signal processing circuit 22 
through the A/D converter 21 . The signal processing circuit 
22 converts the received picture signal into a brightness 
signal (Y) , a color signal (C) , and a video signal and sends 
these signals as picture signals to the TV monitor 13 and 
the video capture portion 29 of the computer 1 . 
[0022] 

The lens block portion 15 of the camera portion 
11 drives the zoom lens 16 and thereby varies the angle of 
view of an object to be photographed. The lens block portion 
15 causes the zoom lens motor 18 that is for example a stepping 
motor torotate , therebydrivingthe zoomlens 16 corresponding 
to a drive command received from the camera controller 24 
of thepantilterportion!2 . The camera controller 2 4per forms 
alenscontrollingoperation (forexample, f ocusingoperat ion 
and zooming operation) , an exposure controlling operation 
(for example, diaphragm controlling operation, gain 
controlling operation, and speed controlling operation of 
electronic shutter) , white balance controlling operation, 
a picture quality controlling operation, and so forth of 
the camera portion 11. In addition, the camera controller 
24 interfaces with the mode controller 23 . As interface 
controlling operations with respect to the zoom lens 16, 
the camera controller 24 sends a control signal to the motor 
driver corresponding' to a drive command of the zoom lens 
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16 received from the mode controller 23 so that the zoom 
lens 16 is placed at the position designated by the command. 
Inaddition, the cameracontrollers 24 always sends positional 
information of the zoom lens 16 to the mode controller 23. 
[0023] 

The camera portion 11 is disposed on the pan tilter 
portion 12 that has a degree of freedom that are rotating 
directions of two axes of panandtilt . The pan tilter portion 
12 causes the pan motor 26 and the tilt motor 27 to rotate 
corresponding to a drive command received from the pan tilter 
controller 25, thereby driving a pan head and a tilt head 
of the pan tilter 28. The motors 26 and 27 are composed 
of for example stepping motors. The pan tilter controller 
2 5 sends a control signal to the motor drivers so that the 
panheadand the tiltheadaredriventopos it ions corresponding 
to a pan drive command and a tilt drive command received 
from mode controller 23 . In addition, the pan tilter 
controller 25 always sends positional information of the 
pan head and the tilt head to the mode controller 23. 
[0024] 

The mode controller 23 controls the overall system 
corresponding to the internal states of the camera portion 
11 and the pan tilter portion 12 and the interface information 
received from the outside of the pan tilter camera 3 as will 
be described later. The mode controller 23 is connected 
with for example the computer 1 and RS-232C interface. The 
mode controller 23 sends drive commands received from the 
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computer 1 to the pan tilter controller 25 and the camera 
controller 24 so as to drive the pan tilter 28 and the zoom 
lens 16 of the lens block portion 15. In addition, the mode 
controller 23 sends current positional information received 
from the pan tilter controller 25 and the camera controller 
24 to the computer 1. 
[0025] 

According to the embodiment, the computer 1 is 
used to select a picture photographed by the pan tilter camera 
3 . The computer 1 processes graphics in the operation area 
6 A and the panorama operat ion area 6B displayed on the monitor 
2 and information of a designated position and a clicking 
operation of the pointing device 14 (mouse 8) and sends the 
resultant data to the mode controller 23 . To display apicture 
photographed by a camera portion 11 on the monitor 2 , a video 
capturing portion 29 is used. The video capturing portion 
2 9 allows a video signal received from the camera portion 
11 to be displayed on the monitor 2 with a desired picture 
quality. In addition, the video capturing portion 29 allows 
a picture to be captured in a particular picture format (for 
example, bit map format, still picture JPEG format, moving 
picture JPEG format, or the like) with a particular picture 
qual i ty and tobe stored in the s tor ingport ion 3 0 (for example, 
a hard disk) of the computer 1. 
[0026] 

Next, with reference to Fig. 4, an example of a 
method for generating a panorama picture displayed in the 
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panorama operation area 6B will be described. It should 
be noted that according to the present invention, a panorama 
picture may be generatedby another method. When the panorama 
generation button 6E is pressed, a panorama picture is 
generated. 

[0027] 

Now, it is assumed that the environment in the 
place where the pan t ilter camera 3 is disposed is a spherical 
surface. The spherical surface is referred to as virtual 
spherical surface. In Fig. 4, two adjacent pictures on the 
virtual spherical surf acearecombinedtoonepanoramapicture . 
To generate a panorama picture, as shown in Fig. 4A, the 
pan tilter camera 3 disposed at the center of the sphere 
photographs two adjacent pictures on the virtual spherical 
surface. The pan tilter camera 3 photographs a plane 
perpendicular to the optical axis of the lens thereof. Fig. 
4D shows a situation of which two adjacent pictures on the 
virtual spherical surface are photographed by the pan tilter 
camera 3 and the two pictures are mapped to the plane 
perpendiculartotheopticalaxis . Whentwoadj acentpictures 
are simply combined, they over lap anddis tort at the over lapped 
portion. 

[0028] 

To prevent two adj acent pictures from overlapping 
and distorting, they are mapped to the virtual spherical 
surface as shown in Fig. 4B. Fig. 4E shows a situation of 
whichtwophotographedpictures that areplanesperpendicular 



totheopticalaxisaremappedtothevirtual spherical surface . 
In such a manner, planes perpendicular to the optical axis 
(namely, photographed pictures) are mapped to the virtual 
spherical surface . The mapped pictures are combined in such 
a manner that an over lappedport ion and an unneces sary port ion 
are removed. The picture mapped on the virtual spherical 
surface is normalized with longitude and latitude. Thus, 
a panorama picture as shown in Figs . 4C and 4D is generated. 
[0029] 

Next, a method for generating a panorama picture 
will be described. In this method, as shown in Fig. 5, one 
panoramapictureisgeneratedbycombininglOpictures . First , 
the pan tilter camera 3 (not shown) disposed at the center 
of the sphere photographs 10 pictures. At this point, as 
shown in Fig. 5A, by matching the optical axis of the lens 
of the pan tilter camera 3 to positions denoted by circles, 
the pan tilter camera 3 can obtain pictures 1 to 10. As 
shown in Fig. 5B, the pictures photographed by the pan tilter 
camera 3 arepictures on theplaneperpendicular to the optical 
axis of the lens. The obtained pictures are mapped to the 
virtual spherical surface. Thereafter, as shown in Fig. 
5C, the pictures are normalized with latitude and longitude . 
The pictures are obtained in such a manner that they are 
smoothly combinedwithout abreak . Thereafter, an overlapped 
portionandunnecessaryportionareremoved. Thus , apanorama 
picture of which 10 pictureare smoothly combined is generated. 

Next, with reference to Fig. 6, another method 
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for generating a panorama picture will be described. In 
this method, pixels obtained by the pan tilter camera 3 are 
designated to pixels of a panorama picture normalized with 
latitude and longitude (namely, coordinates (s, t) ) . As 
in the method shown in Fig. 5, when pixels of pictures 
photographed by the pan tilter camera 3 are designated to 
pixels of a panorama picture, part of pixels of the panorama 
picture may not be designated. All pixels of pictures 
photographed by the pan tilter camera 3 should be designated 
to pixels of the panorama picture. The panorama picture 
is composed of pixels calculated for individual coordinate 
points in the following process. Angular coordinates (a, 
P) (seeFig. 6B) onthevirtualsphericalsurfacecorresponding 
to coordinates (s, t) (see Fig. 6) of a panorama picture 
are calculated corresponding to Eq. (1) . 
[0030] 

(a, p) = (a(s), b(t)) ... (1) 

(Eq. (1) will be described later with reference to Fig. 7.) 
[0031] 

As shown in Fig. 6C, coordinate data (£, r\) of the 
obtained picture is calculated with the coordinates (s, t) , 
the angular coordinates (6, <p) of a pan tilter 28, and 
photographing magnification assuming that the wide edge of 
the photographing apparatus is defined as one magnification 
corresponding to Eq. (2) . 
[0032] 

(§, r|) = (f (a, P, 6, <p, y), g(a, p, 0, cp, y) 
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... (2) 

(Eq. (2) will be described later with reference to Fig. 8 . ) 
[0033] 

Corresponding to the above-described equations, 
pixels of the panorama picture are correlated with obtained 
pictures so as to generate a combined picture (namely, the 
panorama picture) . 
[0034] 

Next, with reference to Fig. 7, a method for 
converting coordinates (s, t) of a panorama picture into 
angular coordinates (a, P) on the virtual spherical surface 
will be described. In Fig. 7A, PragMin represents angular 
data at the left edge assuming that the home position of 
the pan tilter 28 (for example, the center in the moving 
range of the pan tilter 28) is 0 (rag) . PragMax represents 
angular data at the right edge assuming that the home position 
of the pan tilter 28 is 0 (reg) . Ny 2 represents a horizontal 
coordinate of the panorama operation area 6B. -Ny 2 / 2 
represents coordinate data at the right edge of the panorama 
operation area 6B. 
[0035] 

To obtain the pan angle a with the coordinate data 
s, since the following relation is satisfied 

(PragMax - a) : (PragMax - PragMin) = (Ny 2 / 2 

- s) : Ny 2 

the pan angle a is expressed as follows. 

a a PragMax - (PragMax - PragMin) x (Ny 2 / 2 - 

23 



s) / Ny 2 

[0036] 

In Fig. 7B, TragMin represents angular data at 
the upper edge assuming that the home position of the pan 
tilter 28 is 0 (rag) . TragMax represents angular data at 
the lower edge assuming that the home position of the pan 
tilter 28 is 0 (rag) . Nz 2 represents a vertical coordinate 
of the panorama operation area 6B. -Nz 2 / 2 represents 
coordinate data at the upper edge of the panorama operation 
arae 6B. Nz 2 / 2 represents coordinate data at the lower 
edge of the panorama operation area 6B . 
[0037] 

To obtain the tilt angle P with the coordinate 
data t, since the following relation is satisfied, 

(TragMax - P) : (TragMax - TragMin) = (Nz 2 / 2 

- t) : Nz 2 

the tilt angle P is expressed as follows. 

P = TragMax - (TragMax - TragMin) x (Nz 2 / 2 - 

t) / Nz 2 

[0038] 

Next, with reference to Fig. 8, the method for 
converting aplane into a spherical surf ace will be described. 
As shown in Fig. 8A, the spatial coordinates of a point (§, 
r|) of a photographed picture orienting the home position 
(theoriginof latitudeandlongitude) areexpressedasf ollows . 

[0039] 
[Equation 1] 
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[0040] 

At this point, the following relations are 

satisfied. 

ki = tan (X I 2y / (Ny / 2) 

k 2 = tan (\i I 2y / (Nz / 2) 
where (Ny, Nz) represent the drive ranges (y direction and 
z direction) of the mouse pointer of the pointing device 
14 (mouse 8) ; (X, \x) represents the horizontal angle of view 
and vertical angle of view at the wide edge ; and y represents 
the current zoom relative magnification (magnification 
information) assuming that the wide edge is one time (x 1) . 
[0041] 

Inaddition, asshowninFig . 8B, athree-dimensional 
rotation matrix is generally expressed as follows. 

[0042] 
[Equation 2] 
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M<t>> 



cose}) 0 -sin<{> 

0 1 0 
sincj) 0 cos(j> 



cose -sine 0 

/? z (e) = sine cose o 

0 0 1 

[0043] 

Since the direction of one point r|) of a 
photographed picture that is panned and tilted by angular 
information (0, cp) from the home position is the same as the 
direction of one point (a, P) apart from the home position, 
the following relation is satisfied. 
[0044] 

R 2 (9) R y (q>) p = 1R Z (a) R y (p) e x 

When the formula is solved with respect to p, the 

following relation is satisfied. 
[0045] 

[Equation 3] 

p= I R y (-<))) R z (a - 6) R y (p) e x 

cos (a - 6) cosc|)Cosp + sinc|)sinp 

= / sin (a - 6) cosp 

-cos (a - 6) sine}) cosp + cose}) si np 

[0046] 

Thereon, 
[0047] 
[Equation 4] 
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p= I 



a 
b 

c 



(3) 



[0048] 

Thus, £ and r| are obtained as follows. 
[0049] 

5 1 = 1 /a 

§ - -ib / kx = -b / k la 

r| = lc / k 2 = c / k 2a 
With the above formula, (£, r|) projected to the photograph 
coordinates can be obtained with coordinate data with an 
10 angle (a, P) from the home position. 

[0050] 

£ = (-sin (a - 0) cos P) / (ki (cos (a - 9) coscp cos 
P + sin cp sin P) ) 

T| = (-cos (a - 0) sin (p cos P + cos 9 sin P) / (k 2 
15 (cos (a - 0) cos cp cos P + sin cp sin p) ) 

Coordinate data (£, r|) on the obtained picture by the pan 
tilter camera 3 can be obtained from angular coordinates 
(a, P) on the virtual spherical surface corresponding to 
coordinate (s, t) of a panorama picture. Thus, a panorame 
2 0 picture can be generated. 

[0051] 

In contrast, coordinate data with an angle (a, 
P) can be obtained with (£, r|) projected to photograph 
coordinates corresponding to the following formula. 
25 [0052] 
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Since 1 = |p| 

a = 1 / V (1 + k x 2 ^ 2 + k 2 2 r| 2 ) 
b = -ki§ / V (1 + k^ 2 + k 2 2 ii 2 ) 
c = k 2 r| / V (1 + k^ 2 + k 2 2 r| 2 ) 
5 where V ( ) represents that the square root of the calculated 

result in ( ) is obtained. 
[0053] 

Form Formula (3) , the following relations are 

satisfied. 

10 a = cos (a - 9) cos <p cos p + sin q> sinP 

b = sin (a - 0) cosP 

c = -cos (a - 9) sin q> cosP+ cos 9 sinP 
[0054] 

Thus, the following relations are satisfied. 
15 a sin q> + c sin 9 = sinP 

tan (a-0) = b / (a cos9- c sin9) 
Thus, the following relations are satisfied. 

P = sin" 1 (sin <p / V (1 + ki 2 £ 2 + k 2 2 ri 2 ) + sin 9 k 2 

Tl / V(l + kx 2 § 2 + k 2 2 Tl 2 ) 

20 a= tan" 1 (-ki \ / (cos <p - k 2 Tisin9) ) +9 

[0055] 

Thus, the pan angle a and the tilt angle P can be 
obtained as follows. 

(a, P) = (f (§, T| f 6, q>, 7, g(§, Ti, 0, (p, y) ) 
25 ... (4) 

[0056] 

If an error is permitted to some extent, (a, P) 
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can be expressed as follows. 
[0057] 

a=9+ [XI y) x (§/ Ny) 
P=(p+ {\i I y) x (T| / Nz) 
In other words, Eq. (4) can be simplified as follows, 
(a, p) = (f (§, 6, y) , g(T], cp,y) ) . . (5) 
[0058] 

Next, with reference to Fig. 9., a method for 
calculating angular information (a, P) of the pan tilter 28 
expressed by Eq. (4) andEq. (5) with positional coordinates 
(£, r|) of the operation area 6A will be described. First 
of all, an example of a method for directly designating a 
desired point in the operation area 6A will be described. 
Assuming that the center of the operation area 6A is defined 
as (0, 0) of relative coordinates as shown in Fig. 9A, the 
positional coordinates (£, r|) of the mouse pointer of the 
mouse 8 in the operation area 6A are obtained. 
[0059] 

Next, another method for designating a desired 
point generated with a desired area in the operation area 
6A will be described. As shown in Fig. 9A, after a start 
point (ml, nl) in a desired area is designated, an end point 
(m2, n2) inthedesiredareaisdesignated. As the coordinates 
at the center of the rectangle generated with these twopoints , 
a desired point (£, r[) is obtained as Eq. (6) . 

(4,11) = ((ml, nl) + (m2, n2)) / 2 ... (6) 
[0060] 
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Fig. 9A shows coordinates of the mouse 8 (pointing 
device 14) in the operation area 6A. In Fig. 9A, the moving 
range (y direction and z direction) of the mouse pointer 
of the mouse 8 in the operation area 6A is denoted by (Nyi, 
NzJ . Angular coordinates (a, (3) of the pan tilter 28 are 
obtained with positional coordinates r|) of the desired 
point (at themousepointerof themouse8) , angular information 
(0, (p) that represents the orientation of the pan tilter 28, 
andmagnif icationinf ormation (y) of the current zoomrelative 
magnification assuming that the wide edge of the zoom lens 
16 is defined as one magnification corresponding to Eq. (4) 
or Eq. (5) . 
[0061] 

The angular coordinates (a, P) shown in Fig. 9B 
are used to place a position designated by the pointing device 
to the center of the photographed screen assuming that the 
home position of the pan tilter 28 is defined as the origin 
of latitude and longitude. 
[0062] 

The coordinates obtained in Fig . 9 may be absolute 
coordinates of the screen of the monitor 2 or relative 
coordinates assuming that the center of the operation area 
6A is defined as (0, 0) . In the coordinates shown in Fig. 
9, coordinates in the pan direction are represented by 
ml, m2, 0, and a and coordinates in the tilt direction are 
represented by r|, nl, n2 , 9, andp. 
[0063] 



Thus, when the mouse pointer of the mouse 8 is 
present in the operation area 6A, the angular information 
(a, P) of the pan tilter 28 is calculated with the angular 
information (9, cp) of the current pan tilter 28 obtained with 
5 received data, the zoom magnification information (y) , and 

the positional information r|) at the mouse pointer of 
the mouse 8 corresponding to Eq. (4) or Eq. (5) so that the 
designated object is placed at the center of the operation 
area 6A. The angular coordinates (a, P) of the pan tilter 

10 28 are converted into internal positional information (PNew, 

TNew) as shown in Figs. 11A and 11B. The resultant internal 
positional information (PNew, TNew) is storedina sendbuf f er 
along with an absolute position drive command of the pan 
tilter 28. In addition, as will be described later, a data 

15 send request flag (FlagSo) is set so that data is sent upon 

occurrence of a timer event. 
[0064] 

Next, with reference to Fig. 10, a method for 
convertingpositional coordinates r|) of the mouse pointer 

2 0 of the mouse 8 in the panorama operation area 6B of the panorama 

picture into angular coordinates (a, P) corresponding to the 
present invention will be described. As with the method 
for directly designating a desired point in the operation 
area 6A, as shown in Fig. 10A, with a method for directly 

25 designating a desired point in the panorama operation area 

6B, positional coordinates T|) at the mouse pointer of 
the mouse 8 can be obtained. 
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[0065] 

Next, another method for designating a desired 
point generated with a desired area in the panorama operation 
area 6B will be described. As shown in Fig. 10A, after a 
start point (ml, nl) of a desired area is designated, the 
end point (m2, n2) of the desired area are designated. 
Corresponding to Eq. (6), a desired point (£, r|) is obtained. 
[0066] 

In Fig. 10A, the moving range (y direction and 
z direction) of themousepointer of the mouse 8 in the panorama 
operationarea6B ( themovingrangeisdef inedasthecoordinates 
of the mouse pointer of the mouse 8 (pointing device 14) 
in the panorama operation area 6B) is represented by (Ny 2/ 
Nz 2 ) . The moving range is limited by the pan tilter limiter 
6D denoted by dotted lines in the panorama operation area 
6B . The pan tilter limiter 6D represents the moving range 
of the optical axis of the lens of the pan tilter camera 
3 . In other words, a point cannot be designated out of the 
pan tilter limiter 6D. Positional coordinates (x, y) in 
the panorama operation area 6B, angle-of -view information 
(s, t) , and angular information (a, P) of the pan tilter 28 
can be obtained with the positional coordinates (£, r|) of 
thedesiredpoint , theangular information (0, q>) representing 
the orientation of the pan tilter 28, and the magnification 
information (y) as the current zoom relative magnification 
assuming that the wide edge of the zoom lens 16 is defined 
as one magnification corresponding to Eq. (7) , Eq. (8) , and 
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Eq. (9). 

[0067] 

(x, y) = (f 0 (9) , g 0 ((p) ) ... (7) 
(s, t) = (fi(y), gi(y)) ... (8) 
(a, p) = (f (§), g(Tj)) ... (9) 

InFig. 10B,positionalcoordinates (x,y) represent 
the current orientation of the pan tilter 2 8 assuming that 
the home position of the pan tilter 28 is defined as the 
originof latitude and longitude . Angle -of -view information 
(s, t) is the current angle of view in the operation area 
6A. Fig. 10B represents the states of the zoom lens and 
the pan tilter in the panorama operation area 6B . 
[0068] 

In Fig. 10C, angular coordinates (a, |3) are used 
to place the position designated by the pointing device to 
the center of the photographed screen assuming that the home 
position of the pan tilter 28 is defined as the origin of 
latitude and longitude. (PragMax, TragMax) and (PragMin, 
TragMin) represent the movingrangeof thepan tilter (namely, 
the range represented by the pan tilter limiter 6D) . Fig. 
10C shows a drive target value in the pan tilter moving range . 
[0069] 

InFig . 10 , coordinatestobeobtainedmaybeabsolute 
coordinates on the screen of the monitor 2 or relative 
coordinates assuming that the center of thepanorama operation 
area 6B is defined as (0,0). In the coordinates , coordinates 
in the pan direction are represented by£, ml, m2, x, s, and 



a and coordinates in the tilt direction are represented by 
r|, nl, n2, y, t, and p. 
[0070] 

Thus, when the mouse pointer of the mouse 8 is 
present inthepanoramaoperationarea6B, angular information 

(a, p) of the pan tilter 28 is calculated with positional 
information (£, T|) at the mouse pointer of the mouse 8 
corresponding to Eq. (9) so that the designated object in 
the operation area 6A is placed at the center of the operation 
area 6A. Angular coordinates (a, P) of the pan tilter 28 
are converted into internal positional information (PNew, 
TNew) of the pan tilter 28 corresponding to the method shown 
in Figs. 11A and 11B. The internal positional information 

(PNew, TNew) of the pan tilter 28 is stored in a send buffer 
along with an absolute position drive command of the pan 
tilter 28. In addition, as will be described later, a data 
send request flag (FlagSo) is set so that data is sent upon 
occurrence of the timer event. 
[0071] 

Next, a method for converting internal positional 
information (p, t) of thepantilter 28 intoangular information 
(a, P) and a method for converting angular coordinates (a, 
P) into internal positional information (PNew, TNew) of the 
pan tilter 28 will be described with reference to Fig. 11. 
In Fig. 11A, PragMin represents angular data at the left 
edge assuming that the home position of the pan tilter 28 
is 0 (reg) . PragMax represents angular data at the right 
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edge assuming that the home position of the pan tilter 2 8 
is 0 (rag) . PdatMin represents internal count data at the 
left edge of thepan tilter controller 25 . PdatMax represents 
internal counter data at the right edge of the pan tilter 
controller 25 . 
[0072] 

To obtain the pan angle 9 with the pan data p, 
since the following relation is satisfied, 

(PragMax -9) : (PragMax - PragMin) = (PdatMax - 
p) : (PdatMax - PdatMin) 
the pan angle9is expressed as follows. 

9= PragMax - (PragMax - PragMin) x (PdatMax - p) 
/ (PdatMax - PdatMin) 
[0073] 

Thus, the pan data p is expressed as follows, 
p = PdatMax - (PragMax -9) x (PdatMax - PdatMin) 
/ (PragMax - PragMin) 
[0074] 

In addition, to obtain the pan data PNew with the 
pan angle a, since the following relation is satisfied, 

(PragMax -a) : (PragMax - PragMin) = (PdatMax - 
p-new) : (PdatMax - PdatMin) 
the pan data PNew is expressed as follows. 

PNew = PragMax - (PragMax -a) x (PdatMax - PdatMin) 
/ (PragMax - PragMin) 
[0075] 

In Fig. 11B, TragMin represents angular data at 



the upper edge assuming that the home position of the pan 
tilter 28 is 0 (rag) . TragMax represents angular data at 
the lower edge assuming that the home position of the pan 
tiler 28 is 0 (rag) . TdatMin represents internal count data 
at the upper edge of the pan tilter controller 25. TdatMax 
represents internal count data at the lower edge of the pan 
tilter controller 25. 
[0076] 

To obtain the tilt angle (p with the tilt data t, 
since the following relation is satisfied, 

(TragMax -<p) : (TragMax - TragMin) = (TratMax - 
t) : (TdatMax - TdatMin) 
the tilt anglecpis expressed as follows. 

(p= TragMax - (TragMax - TragMin) x (TdatMax - t) 
/ (TdatMax - TdatMin) 
[0077] 

Thus, the tilt data t is expressed as follows, 
t = TdatMax - (TragMax -q>) x (TdatMax - TdatMin) 
/ (TragMax - TragMin) 
[0078] 

To obtain the tilt data TNew with the tilt angle 
P, since the following relation is satisfied, 

(TragMax -0) : (TragMax - TragMin) = (TdatMax - 
t-new) : (TdatMax - TdatMin) 

the tilt data TNew is expressed as follows. 

TNew = TragMax - (TragMax -0) x (TdatMax - TdatMin) 
/ (TragMax - TragMin) 



[0079] 

Next, with reference to Figs . 12Aandl2B, a method 
for converting positional coordinates (£, r|) in the panorama 
operation area 6B into angular coordinates (a, P) of the pan 
tilter 28 and a method for converting angular information 
(0, (p) of the pan tilter 28 into positional coordinates (x, 
y) in the panorama operation area 6B will be described. In 
Fig. 12A, PragMin represents angular data at the left edge 
assuming that the home position of the pan tilter 28 is 0 
(rag) . PragMax represents angular data at the right edge 
assuming that the home position of the pan tilter 28 is 0 
(rag) . Ny 2 representsahorizontal coordinateof thepanorama 
operation area 6B . -Ny 2 / 2 represents coordinate data at 
the left edge of the panorama operation area 6B . Ny 2 / 2 
represents coordinate data at the right edge of the panorama 
operation area 6B. 
[0080] 

To obtain the pan angle a with the coordinate data 
since the following relation is satisfied, 

(PragMax -a) : (PragMax - PragMin) = (Ny 2 / 2 - 

k) s Ny 2 

the pan angle a is expressed as follows. 

<x= PragMax - (PragMax - PragMin) x (Ny 2 / 2 -§) 

/ Ny 2 

[0081] 

To obtain the coordinate data x with the pan angle 
0, since the following relation is satisfied, 



(PragMax -0) : (PragMax - PragMin) = (Ny 2 /0- x) : 

Ny 2 

the coordinate data x is expressed as follows. 

x = Ny 2 /D- (PragMax -0) x Ny 2 / (PragMax - PragMin) 
5 [0082] 

In Fig. 12B, TragMin represents angular data at 
the upper edge assuming that the home position of the pan 
tilter 28 is 0 (rag) . TragMax represents angular data at 
the lower edge assuming that the home position of the pan 
10 tilter 28 is 0 (rag) . Nz 2 represents a vertical coordinate 

of the panorama operation area 6B. -Nz 2 / 2 represents 
coordinate data at the upper edge of the panorama operation 
area 6B. Nz 2 / 2 represents coordinate data at the lower 
edge of the panorama operation area 6B . 
15 [0083] 

To obtain the tilt angle P with the coordinate 
data t|, since the following' relation is satisfied, 

(TragMax -0) : (TragMax - TragMin) = (Nz 2 / 2 - 

T|) : Nz 2 

20 the tilt angle P is expressed as follows. 

P= TragMax - (TragMax - TragMin) x (Nz 2 / 2 -t\) 

/ Nz 2 

[0084] 

To obtain the coordinate data y with the tilt angle 
25 q>, since the following relation is satisfied, 

(TragMax - q>) : (TragMax - TragMin) = (Nz 2 / 2 

- y) : Nz 2 
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the coordinate data y is expressed as follows. 

y = Nz 2 / 2 - (TragMax - 0) xNz 2 / (TragMax - TragMin) 
[0085] 

Next, with reference to Fig. 13, a method for 
converting angle-of -view information (\|/, go) captured by the 
pan tilter 28 into angle-of -view information (s, t) of the 
frame 6C in the panorama operation area 6B will be described. 
Fig. 13A shows the current angle-of -view information 
co) of the pan tilter 28. The angle-of -view information (\|/, 
co) is expressed as follows. 

(\|/, co) = 1 / y x (\|/0, ooO) 
At this point, (\[/0 , ooO) represent the horizontal angle of 
viewandtheverticalangleof viewat thewideedge . y represents 
the magnification of the lens assuming that the wide edge 
is defined as one magnification. 
[0086] 

As shown in Fig. 13B, PragMin represents angular 
data at the left edge assuming that the home position of 
the pan tilter 28 is 0 (rag) . PragMax represents angular 
data at the right edge assuming that the home position of 
the pan tilter 28 is 0 (rag) . Ny 2 represents a horizontal 
coordinate of the panorama operation area 6B. -Ny 2 / 2 
represents coordinate data at the left edge of the panorama 
operation area 6B . Ny 2 / 2 represents coordinate data at 
the right edge of the panorama operation area 6B . 
[0087] 

To obtain the horizontal angle of view s with the 

39 



horizontal angle of view \|/, since the following relation 
is satisfied, 

V|/ : (PragMax - PragMin) = s : Ny 2 
horizontal angle of view s is expressed as follows. 

s = \|/ x Ny 2 / (PragMax - PragMin) 
[0088] 

In Fig. 13C, TragMin represents angular data at 
the lower edge assuming that the home position of the pan 
tilter 28 is 0 (rag) . TragMax represents angular data at 
the upper edge assuming that the home position of the pan 
tilter 28 is 0 (rag) . Nz 2 represents a vertical coordinate 
of the panorama operation area 6B . -Nz 2 / 2 represents 
coordinate data at the lower edge of the panorama operation 
area 6B. Nz 2 / 2 represents coordinate data at the upper 
edge of the panorama operation area 6B . 
[0089] 

To obtain the vertical angle of view t with the 
vertical angle of view go, since the following relation is 
satisfied, 

co : (TragMax - TragMin) = t : Nz 2 
the vertical angle of view t is expressed as follows, 
t = co x Nz 2 / (TragMax - TragMin) 
[0090] 

Thus, the angle-of -view information (s, t) shown 
in Fig. 13D is displayed as the frame 6C in the panorama 
operation area 6B. 
[0091] 



Next, with reference to Fig. 14, a method for 
converting the positional information (z) of the zoom lens 
16 into magnification information (y) will be described. 
In Fig. 14, the vertical axis represents information of lens 
magnification, whereas the horizontal axis represents the 
internal information of zoom lens. The positional 
information (z) of the zoom lens 16 is converted into the 
magnification information (y) by the computer 1 corresponding 
to a conversion graph shown in Fig. 14. For example, the 
positionalinf ormation (z) isconvertedintothemagnif ication 
information (y) corresponding to a ROM table or an equation. 
[0092] 

Next, with reference to Fig. 15, an example of 
a control algorithm of the computer 1 will be described. 
When the program is executed, the flow advance to step SI. 
At step SI, the operation area 6, the panorama operation 
area 6B, the cursor 7, and the pan tilter limiter 6D are 
initialized and displayed on the monitor 2 as shown in Fig. 
2. The range of the pan tilter limiter 6D may be fixed or 
variable. At step S2, a timer is set so that the computer 
1 communicates with the mode controller 23 at predetermined 
intervals. After such initial setup operations have been 
completed, the flow advances to step S3. At step S3, the 
system waits for an occurrence of an event. Corresponding 
to an event that occurs, the flow advances to a relevant 
step (for example, a timer event (at step S4) , a mouse button 
down event (at step S5) , a mouse button up event (at step 



S6) , and a mouse move event (at step S7) ) . 
[0093] 

Next , with reference to a flow chart shown in Fig . 
16, the algorithm of the timer event will be described. The 
t imer event is anevent for causing the computer It o communicate 
with the mode controller 23 at predetermined intervals . The 
timer event occurs at intervals of for example 50 msec . When 
the timer event occurs, the flow advances to step Sll. At 
steps 11, the system determines whether or not a communication 
port has been set. When the communication port has been 
set (namely, the determined result at step Sll is Yes) , the 
flow advances to step S12 . When the communication port has 
not been set (namely, the determined result at step Sll is 
No) , the flow advances to step S18. At the first time the 
communication port has not been set, the flow advances to 
step S18. At step S18, the system opens the communication 
port. Actually, at step S18, the system opens an RS-232C 
port of the computer 1. Thereafter, the flow advances to 
step S16. 

[0094] 

Thereafter, inthe timer event , the systemperf orms 
a receive data checking process, an analyzing process, a 
data sending process for data stored in the send buffer (such 
as the drive command for the pan tilter 28) , and/or a 
communication data sending process for state check requests 
for the pan tilter 2 8 and the zoom lens 16 . In this algorithm, 
the flow advances from step Sll to step S12 . At step S12, 
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the system determines whether or not data is stored in the 
receive buffer. When data is stored in the receive buffer 
(namely, the determined result at step S12 is Yes) , the flow 
advances to step S13 . When data is not stored in the receive 
buffer (namely, the determined result at step S12 is No) , 
thef low advances to step S14 . At stepS13 , the systemanalyzes 
receivedatastoredinthereceivebuf f erandobtainspositional 
information (p, t) of the pan tilter 28 and positional 
information (z) of the zoom lens 16 that have been requested 
to the mode control 1 er 2 3 . The system convert s the pos i t ional 
information (p, t) of the pan tilter 28 and the positional 
information (z) of the zoom lens 16 into angular information 
(0, q>) of the pan tilter 28 and magnification information 
(y) of the zoom lens 16 corresponding to methods shown in 
Fig. 11 and 14. 
[0095] 

At step S14 , the system determines whether or not 
a data send request has been issued. When a data send request 
has been issued (FlagSo == True) (namely, the determined 
result at step S14 is Yes) , the flow advances to step S19. 
At step S19, the system sends data stored in the send buffer 
and resets the send request flag (FlagSo == False) . Next, 
the flow advances to step S16. An example of data stored 
in the send buffer is data of a drive command of the pan 
tilter 28 designated with the mouse 8. When a send request 
has not been issued (FlagSo == False) (namely, the determined 
result at step S14 is No) , the flow advances to step S15. 
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At step S15, the system sends position request commands for 
the pan tilter 2 8 and the zoom lens 16 from the computer 
1 to the mode controller 23. 
[0096] 

At step S16 , the system compares the oldpositional 
information of the pan tiler 28 with the new positional 
information thereof and determines whether or not the 
positional information (p, t) hasvaried. Whenthepositional 
information (p, t) of the pan tilter 28 has varied (namely, 
the determined result at step S16 is Yes) , the flow advances 
to step S20. When the positional information (p, t) of the 
pan tilter 28 has not varied (namely, the determined result 
at step S16 is No) , the flow advances to step S17 . At step 
S17, the system compares the oldpositional information of 
the zoom lens 16 with the new positional information thereof 
and determines whether or not the positional information 

(z) has varied. When the positional information (z) of the 
zoom lens 16 has varied (namely, the determined result at 
step S17 is Yes) , the flow advances to step S20. When the 
positional information (z) of the zoom lens 16 has not varied 

(namely, the determined result at step S17 is No) , this event 
is completed. 
[0097] 

At step S20, when the positional information (p, 
t) of the pan tilter 28 and/or the positional information 
(z) of the zoom lens 16 has varied, the system redraws the 
frame 6C in the panorama operation area 6B. At this point, 
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the system converts the positional information (p, t) of 
the pan tilter 28 into the angular information (0, <p) . In 
addition, the system converts the positional information 
(z) of the zoom lens 16 into the magnification information 
(y) . With the converted angular information (0, <p) and 
magnification information (y) , the system calculates 
positional coordinates (x, y) of the pan tilter 28 and 
angle-of -view information (s, t) that is the angle of view 
displayed in the operation area 6A corresponding to Eq. (7) 
and Eq. (8) , respectively. Corresponding to the resultant 
positional coordinates (x, y) and angle-of -view information 
(s, t) , the system draws the frame 6C in the panorama operation 
area 6B. 

[0098] 

At step S16 , the system compares the oldpositional 
information (p, t) of the pan tilter 2 8 with the newpositional 
information (p, t) thereof. Alternatively, the system may 
compare the old angular information (9, cp) of the pan tilter 
28 with the new angular information (0, cp) thereof. In this 
case, at step S20, with the new angular information (0, cp) , 
the system calculates the positional coordinates (x, y) 
corresponding to Eq. (7) . Likewise, atstepS17, the system 
compares the oldpositional information (z) of the zoom lens 
16 with the new positional information (z) thereof. 
Alternatively, the system may compare the old magnification 
information (y) of the zoom lens 16 with the new magnificat ion 
information (y) thereof. Inthiscase, atstepS20, the system 
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calculates the angular information (s, t) with the new 
magnification information (y) corresponding to Eq. (8) . 
[0099] 

Next, with reference to a flow chart shown in Fig. 
17, the algorithm of the mouse move event will be described. 
The mouse move event is an event that occurs when the mouse 
8 is moved. According to the present invention, the mouse 
move event is used to select a drive position of the pan 
t ilter 28 . When the mouse move event occurs , the flow advances 
to step S21. At step S21, the system determines whether 
or not the mouse pointer of the mouse 8 is present in the 
operation area 6A, the panorama operation area 6B, or the 
other area shown in Fig. 2. When the mouse pointer of the 
mouse 8 is present in the operation area 6A (namely, the 
determined result at step S21 is Yes) , the flow advances 
to step S22. When the mouse pointer of the mouse 8 is not 
present in the operation area 6A (namely, the determined 
result at step S21 is No) , the flow advances to step S24 . 
At step S22 , the system sets an operation area flag (Flag-rin 
= = True) and clears a panorama operation area flag (Flag-pin 
== False) . 

[0100] 

At step S24, since the mouse pointer of the mouse 
8 is not present in the operation area 6A, the system clears 
the operation area flag (Flag-rin == False) . At step S25, 
the system determines whether or not the mouse pointer of 
the mouse 8 is present in the panorama operation area 6B . 



When the mouse pointer of the mouse 8 is present in the panorama 
operation area 6B (namely, the determined result at step 
S25 is Yes) , the flow advances to step S26. When the mouse 
pointer of the mouse 8 isnot present in the panorama operation 
area 6B (namely, the determined result at step S25 is No) , 
the flow advances to step S27. At step S26, the system sets 
the panorama operation area flag (Flag-pin == True) . At 
step S27 , since the mouse pointer of the mouse 8 is not present 
inthepanoramaoperationarea6B, thesystemclear thepanorama 
operation area flag (Flag-pin == False) . 
[0101] 

When the mouse pointer of the mouse 8 is present 
in the operation area 6A or the panorama operation area 6B 
(namely, the determined result at step S21 or step S25 is 
Yes) , at stepS23, the system obtains posit ional coordinates 
T]) of the mouse pointer of the mouse 8 assuming that the 
center of the operation area is defined as (0, 0) of relative 
coordinates . 
[0102] 

In this flow chart, when the mouse pointer of the 
mouse 8 is present in the operation area 6A (namely, the 
determined result at step S22 is Yes) , the system sets the 
operationareaf lag (Flag-rin==True) . Whenthemousepointer 
of the mouse 8 is not present in the operation area 6A (namely, 
the determined result at step S22 is No) , the system clears 
the operation area flag (Flag-rin == False) . When the mouse 
pointer of the mouse 8 is present in the panorama operation 

47 



area 6B (namely, the determined result at step S25 is Yes) , 
the system sets the panorama operation area flag (Flag-pin 
= = True) . When the mouse pointer 8 is not present in the 
panorama operation area 6A (namely, the determined result 
at step S25 is No) , the system clears the panorama operation 
area flag (Flag-pin == False) . When the mouse pointer of 
the mouse 8 is present in the operation area 6A or the panorama 
operation area 6B (namely, the determined result at step 
S21 or S35 is Yes) , the system designates the positional 
coordinates of the mouse pointer of the mouse 8 to (§, T|) 
assuming that the center of each operation area is defined 
as (0, 0) of relative coordinates. 
[0103] 

Next, the mouse button down event and the button 
up event will be described. In the method for directly 
designating a desired point of the operation area 6A or the 
panorama operation area 6B, only the algorithm of a mouse 
button down event shown in Fig. 18 is used. In the method 
for designating a desired point generated with a desired 
area, both the algorithm of a mouse button down event shown 
in Fig. 19 and the algorithm of a mouse button up event shown 
in Fig. 20 are used. 
[0104] 

With reference to a flow chart shown in Fig. 18, 
the algorithm of the button down event for the method for 
directly designating a desired point of the operation area 
will be described. This event is an event that occurs when 
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the left button of the mouse 8 is pressed. In the present 
invention, this event is used as trigger information for 
driving the pan tilter 28. When this event occurs, the flow 
advances to step S31. At step S31, the system determines 
whether or not the mouse pointer 8 is present in the operation 
area 6A corresponding to the operation area flag. When the 
operation area flag has been set (FlagRin == True) (namely, 
the determined result at step S31 is Yes) , since the mouse 
pointer of the mouse 8 is present in the operation area 6A, 
the flow advances to step S32 . When the operation area flag 
has been cleared (FlagRin == False) (namely, the determined 
result at step S31 is No) , since the mouse pointer of the 
mouse 8 is not present in the operation area 6A, the flow 
advances to step S34 . 
[0105] 

When the mouse pointer of the mouse 8 is present 
in the operation area 6A (namely, the determined result at 
step S31 is Yes) , the flow advances to step S32. At step 
S32, the system calculates angular information (a, P) of the 
pantilter 28 with theangular information (0, (p) of the current 
pantilter2 8obtainedf romthereceiveddata, themagnif icat ion 
information (y) of the zoom lens 16, and the positional 
coordinate (£, r|) of the mouse pointer of the mouse 8 in the 
operation area 6A corresponding to Eq. (4) or Eq. (5) so 
that the designated object in the operation area is placed 
at the center of the screen. 
[0106] 



At step S33, the system converts the angular 
information (a, 0) of the pan tilter 28 into the internal 
positional information (PNew, TNew) corresponding to the 
method shown in Fig. 11. The system stores the converted 
positional information (PNew, TNew) in the send buff er along 
with the absolute position drive command of the pan tilter 
28 . In addition, the system sets the data send request flag 
(FlagSo == True) and sends the data with the process of the 
timer event. 
[0107] 

After the system has determined that the mouse 
pointer of the mouse 8 is not present in the operation area 
6A (namely, the determined result at step S31 is No) , the 
flow advances to step S34 . At step S34 , the system determines 
whether or not the mouse pointer of the mouse 8 is present 
inthepanoramaoperationarea6B corresponding to thepanorama 
operation area flag. When the panorama operation flag has 
been set (FlagPin == True) (namely, the determined result 
at step S34 is Yes) , since the mouse pointer of the mouse 
8 of the panorama operation area 6B is present in the panorama 
operation area 6B, the flow advances to step S35. When the 
panorama operation flag has been cleared (FlagPin == Fale) 
(namely, the determined result at step S34 is No) , this event 
is completed. 
[0108] 

In this flow chart , the system determines whether 
or not the mouse pointer of the mouse 8 is present in the 



operation area 6A or the panorama operation area 6B 
corresponding to the operation area flag (FlagRin) and the 
panorama operation area flag (FlagPin) . When the mouse 
pointer of the mouse 8 is not present in the operation area 
6A and the panorama operation area 6B, this event becomes 
invalid. 

[0109] 

When the mouse pointer of the mouse 8 is present 
in the panorama operation area 6B (namely, the determined 
result at step S34 is Yes) , the flow advances to step S35. 
At step S35, the system calculates angular information (a, 
P) of the pan tilter 28 with the positional information (§, 
r|) at the mouse pointer of the mouse 8 in the panorama operation 
area 6B corresponding to Eq. (9) so that the designated obj ect 
in the operation area is placed at the center of the screen. 
Thereafter, the flow advances to step S33 . 
[0110] 

Next, with reference to Figs. 19 and 20, the 
algorithms of the button down event and the button up event 
for the method for designating a desired point generated 
with a desired area in the panorama operation area 6B will 
be described, respectively. 
[0111] 

With reference to the flow chart shown in Fig. 
19, the algorithm of the button down event will be described. 
This event is an event that occurs when the left button of 
the mouse 8 is pressed. In this embodiment, this event is 



used as an event for determining the start point of a desired 
area. When this event occurs, the flow advances to step 
S41. At step S41, the system determines whether or not the 
mouse pointer of the mouse 8 is present in the operation 
5 area 6A corresponding to the operation area flag (FlagRin) . 

When the operation area flag has been set (FlagRin == True) 
(namely, the determined result at step S41 is Yes) , since 
the mouse pointer of the mouse 8 is present in the operation 
area 6A, the flow advances to step S42 . When the operation 
10 area flag has been cleared (FlagRin == False) (namely, the 

determined result at step S41 is No) , since the mouse pointer 
of the mouse 8 is not present in the operation area 6A, the 
flow advances to step S44 . 
[0112] 

15 When the mouse pointer of the mouse 8 is present 

in the operation area 6A (namely, the determined result at 
step S41 is Yes) , at step S42, the system sets an operation 
areastartpointobtainf lag (FlagRstart = True) . Thereafter, 
the flow advances to step S43 . At step S43 , the system stores 

20 positional coordinates (ml, nl) at which the left button 

of the mouse 8 is pressed as the start point of the desired 
area . 

[0113] 

After the system has determined that the mouse 
25 pointer of the mouse 8 is not present in the operation area 

6A, at step S44, the system determines whether or not the 
mousepointerof themouse8 ispresent in thepanorama operation 
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area 6B corresponding to the panorama operation area flag 
(FlagPin) . When the panorama operation area flag has been 
set (FlagPin == True) (namely, the determined result at step 

544 is Yes) , since the mouse pointer of the mouse 8 is present 
in the panorama operation area 6B, the flow advances to step 

545 . When the panorama operation area flag has been cleared 
(FlagPin == False) (namely, the determined result at step 
S44 is No) , this event is completed. 

[0114] 

In this flow chart , the system determines whether 
or not the mouse pointer of the mouse 8 is present in the 
operation area 6A or the panorama operation area 6B 
corresponding to the operation area flag (FlagRin) and the 
panorama operation area flag (FlagPin) . When the mouse 
pointer of the mouse 8 is not in the operation area 6A and 
the panorama operation area 6B, this event becomes invalid. 
[0115] 

When the mouse pointer of the mouse 8 is present 
in the panorama operation area 6B (namely, the determined 
result at step 44 is Yes) , the flow advances to step S45. 
At step S45, the system sets a panorama operation area start 
point obtain flag (FlagPstart) . Thereafter, the flow 
advances to step S43 . 
[0116] 

Next, with reference to a flow chart shown in Fig. 
20, the algorithm of the button up event will be described. 
This event is an event that occurs when the left button of 
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the mouse 8 is released. In the present invention, the button 
up event is used as an event for determining the end point 
of a desired area. 
[0117] 

When this event occurs, the flow advances to step 
S51. At step S51, the system determines whether or not the 
operation area flag has been set (FlagRin == True) (namely, 
the mouse pointer of the mouse 8 is present in the operation 
area 6A) . When the operation area flag has been set (FlagRin 
= True) (namely, the determined result at step S51 is Yes) , 
since the mouse pointer of the mouse 8 is present in the 
operation area 6A, the flow advances to step S52 . When the 
operationareaf laghasbeencleared (FlagRin==False) (namely, 
the determined result at step S51 is No) , since the mouse 
pointer of the mouse 8 is not present in the operation area 
6A, the flow advances to step S57. At step S52, the system 
determines whether or not the left button of the mouse 8 
has been pressed in the operation area 6A corresponding to 
anoperationarea start point obtainf lag (FlagRstart) . When 
the start point obtain flag has been set (FlagRstart == True) 
(namely, the determined result at step S52 is Yes) , since 
the lef t buttonof the mouse 8 hasbeenpressedin the operation 
area 6A, the flow advances to step S53 . When the start point 
obtain flag has been cleared (FlagRstart == False) (namely, 
the determined result at step S52 is No) , since the left 
button of the mouse 8 has not been pressed in the operation 
area 6A, the flow advances to step S57. 
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[0118] 

In other words, at steps S51 and S52, the system 
determines whether or not the operation area flag and the 
operationarea st art point obtain f laghavebeen set or cleared. 
When the operation area flag and the start point obtain flag 
have been set (FlagRin == True and FlagRstart == True) , the 
system determines that the drive command has taken place 
in the operation area 6A. Otherwise, at steps S57 and S58, 
the system determines whether or not the panorama operation 
area flag (FlagPin) and the panorama operation area start 
point obtain flag (FlagPstart) have been set or cleared. 
[0119] 

When the drive command has taken place in the 
operationarea (namely, the operation area flag and the start 
point obtain flag have been set ( FlagRin = = True and FlagRstart 
== True), at step S53, the system stores the positional 
coordinates (m2, n2) at which the left button of the mouse 
8 has been released in the operation area 6A as the end point 
of the desired area. Thereafter, the system calculates 
positional information r|) as the coordinates of the center 
of therectangl eareageneratedwiththepos it ional coordinates 
(ml, nl) of thestartpointof thedesiredareaandthepositional 
coordinates (m2, n2) of the end point thereof. 
[0120] 

At step S54, the system calculates angular 
information (a, P) of the pan tilter 28 with the angular 
information (0, q>) of the pan tilter obtained from the received 
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data, the magnification information (y) of the zoom lens 
16, and the positional information (£, r|) at the mouse pointer 
of the mouse 8 corresponding to Eq. (4) or Eq. (5) . 
[0121] 

At step S55, the system converts the angular 
information (a, |3) of the pan tilter 28 into the internal 
positional information (PNew, TNew) of the pan tiler 28 
corresponding to the method shown in Fig. 11 and stores the 
positional information (PNew, TNew) to the send buff er along 
with the absolute position drive command. In addition, the 
system sets the data send request flag (FlagSo == True) and 
sends data with the process of the timer event. 
[0122] 

At step S56, after the system has checked the mouse 
button up event in each operation area, the system clears 
the operation area start point obtain flag and the panorama 
operation area start point obtain flag (FlagRstart == False 
and FlagPstart == False) . Thereafter, this event is 
completed. 

[0123] 

At step S57, the system determines whether or not 
the mouse pointer of the mouse 8 is present in the panorama 
operation area 6B corresponding to the panorama operation 
area flag (FlagPin) . When the panorama operation area flag 
has been set (FlagPin == True) (namely, the determined result 
at step S57 is Yes) , since the mouse pointer of the mouse 
8 is present in the panorama operation area 6B, the flow 

56 



advances to step S58 . When the panorama operation area flag 
has not been set (FlagPin == False) , since the mouse pointer 
of the mouse 8 is not present in the panorama operation area 
6B, the flow advances to step S56. At step S58, the system 
determines whether or not the left button of the mouse 8 
hasbeenpressedinthepanoramaoperationarea6Bcorresponding 
to the panorama operation area start point obtain flag 
(FlagPstart) . When the start point obtain flag has been 
set (FlagPstart == True) (namely, the determined result at 
step S58 is Yes) , since the left button of the mouse 8 has 
been pressed in the panorama operation area SB, the flow 
advances to step S59. When the start point obtain flag has 
not been set (FlagPstart == False) (namely, the determined 
result at step S58 is No) , since the left button of the mouse 
8 has not been pressed in the panorama operation area 6B, 
the flow advances to step S56. 
[0124] 

When the panorama operation area flag and the 
panorama operation start point obtain flag have been set 
(FlagPin == True and FlagPstart == True) at steps S57 and 
S58, the system determines that a drive command has issued 
in the panorama operation area 6B . When the conditions at 
steps S51, S52, and S58 are not satisfied, this event becomes 
invalid. 

[0125] 

Whenthedrivecommandhasbeenissuedinthepanorama 
operation area 6B (namely, the panorama operation area flag 
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and the start obtain flag have been set (FlagPin == True 
and Flag-pstart == True), the flow advances to step S59. 
At step S59 , the system stores the positional coordinates 
(m2, n2) at which the left button of the mouse 8 has been 
released in the panorama operation area 6B as the end point 
of the desired area. The system calculates the positional 
information T]) of the mouse pointer of the mouse 8 as 
the coordinates of the center of the rectangle area with 
the positional coordinates (ml, nl) of the start point of 
the desired area that has been stored and the positional 
coordinates (m2, n2) of the end point of the desired area 
corresponding to Eq. (6) . 
[0126] 

At step S60, the system calculates angular 
information (a, (3) of the pan tilter 28 with the positional 
information r|) at the mouse pointer of the mouse 8 in 
the panorama operation area 6B corresponding to Eq. (9) so 
that the designated object in the panorama operation area 
is placed at the center of the screen. Thereafter, the flow 
advances to step S55. 
[0127] 

In the above-described embodiment, one computer 
performs all processes of the system. On the other hand, 
according to a second embodiment of the present invention, 
as shown in Fig . 21, processes are shared by a server computer 
and a client computer so as to control a pan tiler camera 
through a network that has a restriction of a communication 
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capacity. In Fig. 21, a computer 1 is connected to a monitor 
2 and a mouse 8. The computer 1 controls the operation of 
a pan tilter camera 3 disposed at a remote place through 
a transmission line and a server 9. In other words, the 
computerlcomposesacontrollerf oraphotographingapparatus . 
The transmission line may be a communication line (radio 
communication line or a cable communication line) , a network, 
or the like. The computer 1 has a relation of a client to 
the server 9. A plurality of computers 1 can be connected 
to the server 9. 
[0128] 

The pan tilter camera 3 and the server 9 are disposed 
on a real scene in an environment denoted by reference numeral 

4 . A screen photographed by the pan tilter camera 3 disposed 
on the real scene 4 is denoted by reference numeral 5. 
Hereinafter, the screen 5 is referred to as photographed 
screen . Thephotographedscreen5 isanactuallyphotographed 
screen. When the zoom lens is placed on the telephotograph 
side, the angle of view decreases. In contrast, when the 
zoom lens is placed on the wide-angle side, the angle of 
view increases . 

[0129] 

A picture photographed by the pan tilter camera 

5 is sent to a server 9 . The server 9 converts the photographed 
picture into video data . The video data is sent to the computer 
1 through a transmission line. The video data sent to the 
computer 1 is decoded and displayed on the monitor 2 . The 
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monitor 2 displays the photographed screen 5 in the operation 
area 6A thereof. A panorama picture with which a picture 
photographed by the pan tilter camera 3 is superimposed is 
displayed in the panorama operation area 6B . As with the 
above-described embodiment , a desired point of the panorama 
operation area 6B (or the operation area 6A) or a desired 
point generated with a desired point is designated with the 
mouse 8 (cursor 7) . The pan tilter camera 3 is driven through 
the server 9 and the transmission line and thereby the 
photographed screen is moved . In other words , the pan tilter 
camera 3 is controlledthrough the server 9 so that the selected 
object is placed at the center of the operation area 6A. 
[0130] 

Fig. 22 is a block diagram showing the overall 
system of the second embodiment of the present invention. 
Since the structures and functions of the camera portion 
11 and the pan tilter portion 12 are the same as those of 
the first embodiment, the structures thereof are omitted 
in Fig. 22. The server 9 comprises a controlling portion 
131, a video capture portion 129, and a storing portion 130 . 
The video capture portion 129 is composed of a video capture 
board. The computer 1 is connected to a transmission path 
132 through a network. The computer 1 is composed of a 
controllingportion3 landsof orthaswiththef irstembodiment . 
Since the algorithms used in the computer 1 are the same 
as those of the first embodiment, for simplicity, their 
description is omitted. 
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[0131] 

Rays emitted from an object are sent to the camera 
portion 11 as with the first embodiment . The camera portion 
11 converts the rays into various signals such as abrightness 
signal (Y) , a color signal (C) , anda video signal and supplies 
the resultant signals as picture signals to a TV monitor 
13 and the video capture portion 129 of the server 9. As 
with the first embodiment, the pan tilter portion 12 has 
a mode controller, a camera controller, and a pan tilter 
controller. These controllers control the camera portion 
11 and the pan tilter 28. The mode controller 23 controls 
the overall system corresponding to the internal states of 
the camera portion 11 and the pan tilter portion 12 and an 
external command as with the first embodiment. 
[0132] 

The mode controller 23 is connected to the server 
9 throughacommunicationpath (inreality, RS232Cinterf ace) . 
The mode controller 23 sends commands received from the server 
9 and commands received from the computer 1 through the server 
9 to the pan tilter controller and the camera controller 
so as to drive the pan tilter and the zoom lens of the lens 
block portion. The mode controller 23 always receives 
information from the pan tilter controller and the camera 
controller so as to send the inner state of the pan tilter 
camera to the outside through the server 9. 
[0133] 

The server 9 obtains the inner state of the pan 
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tiltercamera (for example, thecurrentpositional information 
of the pan tilter and the zoom lens, and so forth) from the 
mode controller 2 3 of thepan tilt erport ion 12 at predetermined 
intervals. To send a picture photographed by the camera 
portion 11 to the transmission path 132, the video capture 
portion 129 is used. The video capture portion 129 converts 
apicture signal receivedf rom the cameraport ion 11 todigital 
picture data that is sent to the transmission path 132 in 
any quality (in the present embodiment, still picture JPEG 
f ormat or st illpicturebit map format ) . Theresultant digital 
picture is stored in a storing portion 130 (for example, 
a hard disk) . 
[0134] 

When the computer 1 issues a connection request 
to the server 9, the server 9 sends a GUI (Graphical User 
Interface) panel information to the computer 1 so as to display 
a picture on the monitor 2. The panel information is an 
arrangement of a panel and a program that runs on the computer 
1 when the mouse is operated on the panel . Examples of the 
panel information are programs written in HTML, JAVA, and 
so forth. Picture data photographed by the pan tilter camera 
and the state thereof are sent to the computer 1 through 
the transmission path 132 at predetermined intervals. 
[0135] 

In another embodiment, Internet is used as the 
transmission path 13 2 . Data is exchanged on the transmission 
path 132 using the HTTP protocol. The computer 1 causes 
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the monitor 2 to display GUI panel information, picture 
information, the state of the pan tilter camera, and so forth 
received from the server 9 with an Internet browser. An 
operation area 6A, a panorama operation area 6B, a panorama 
picturegenerationbutton6E, zoomoperationbuttons , acursor 
7 of a pointing device 14 (mouse 8) , and so forth displayed 
on the GUI panel of the monitor 2 . Picture data received 
from the server is decoded and displayed in the operation 
area 6A. When the picture data is updated, the picture is 
also rewritten in the operation area 6A. The moving range 
of the pan tilter camera, the position of the pan tilter, 
angle-of -view of the zoom, and so forth are displayed on 
the panorama operation area 6B, with the same method as the 
first embodiment. The computer 1 executes the operation 
program for the GUI panel received from the server 9. 
[0136] 

In the second embodiment of the present invention, 
a drive command of the pan tilter camera 3 and an operation 
command of the server 9 are generated with a clicking operation 
of the mouse 8 . When the mouse 8 is clicked on the panorama 
generation button 6E, the computer 1 causes the server 9 
to generate a panorama screen. When the server 1 receives 
this command, as with the first embodiment, the server 9 
moves the pan tilter and the zoom lens to relevant positions , 
photographs ten pictures at these positions, maps them to 
the virtual spherical surface, normalizes them with latitude 
and longitude, and combines them. After the server 9 has 
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combined these pictures as a panorama picture, it converts 
the panorama picture into a JPEG format picture . The servers 
9 sends the resultant picture to the computer 1 through the 
transmission line 132 . 
[0137] 

The computer 1 displays the received panorama 
picture in the panorama operation area 6B of the monitor 
2. Thus, the user can see the environment at the position 
of the pan tiler camera 3 at a glace. When the mouse 8 is 
clicked in the panorama operation area 6B, the computer 1 
sends to the server 9 a command (absolute position drive 
command) that causes the posit ion at which the mouse is clicked 
on the panorama picture to be placed at the center of the 
operation area 6A (picture) . The server 9 sends this command 
to the pan tilter camera 3. Thus, the pan tilter is driven 
to a relevant position. In such a manner, the drive target 
of the pan tilter is designated on the panorama screen. 
Consequently, the user can easily operate the pan tilter 
without need to consider a drive command on the network, 
a delay of a video signal, and so forth. 
[0138] 

In the first embodiment, whenever the pan tilter 
camera 3 sends a picture to the computer 1, the computer 
1 combines it and displays the combinedpicture in the panorama 
operation area 6B . Alternatively, after the computer has 
combined all pictures, it may display the resultant picture 
in the panorama operation area 6B . 
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[0139] 

According to the first embodiment, the operation 
area 6A and the panorama operation area 6B are displayed 
on the monitor 2 connected to the computer 1 . Alternatively, 
the operation area 6A and/ or the panorama operation area 
6B may be displayed on another display unit other than the 
monitor 2 . 

[0140] 

According to the first embodiment , the pan tilter 
camera 3 is driven by operating the operation area 6A and 
thepanoramaoperationarea6Bwiththemouse8 . Alternatively, 
one of the operation area 6A and the panorama operation area 
6B may be operated with the mouse 8. 
[0141] 

According to the first embodiment, the operation 
area 6A and the panorama operation area 6B are displayed 
on the monitor 2 . Alternatively, only the panorama operation 
area 6B may be displayed on the monitor 2 . 
[0142] 

According to the first embodiment, the operation 
area 6A and the panorama operation area 6B are displayed 
on the monitor 2. By operating the operation area 6A and 
the panorama operation area 6B with the mouse 8, the pan 
tilter camera 3 is freely driven. Alternatively, a panorama 
picture may be displayed on the monitor 2. In this case, 
the pan tilter camera 3 may be driven with an oper a t ion port ion 
such as eight-direction keys. 
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[0143] 

According to the above -described embodiments , the 
photographing range of the pan tilter camera 3 may be the 
maximum moving range of the pan tilter camera 3 or limited 
with a limiter . The function for limited the photographing 
range with the limiter may be provided by the pan tilter 
camera 3 or the computer 1. 
[0144] 

Inthe first embodiments , adesiredpoint generated 
with a desired area is placed at the center of thereof. 
Alternatively, a desired point may be placed at for example 
the center of gravity, the incenter, the circumcenter , or 
the orthocenter of the area. 
[0145] 

According to the first embodiment, a panorama 
picture displayed in the panorama operation area 6B is not 
limited as long as it represents the environment in which 
thepantilter camera3 isdisposed. Forexample, thepanorama 
picturemaybeamovingpicture, anintermittentstillpicture, 
or a still picture. 
[0146] 

Accordingtothesecondembodiment , f orsimplicity , 
one computer 1 is connected to the remote server 9 and the 
pan tilter camera 3 that are disposed at a remote place. 
Alternatively, a plurality of servers 9 and a plurality of 
pan tilter cameras 3 maybe disposed worldwide . For example, 
one pan tilter camera 3 may be controlled by a plurality 
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of computers through for example Internet. 
[0147] 

[Effects of the Invention] 

Accordingtothepresent invention, withapanorama 
picture, the user can see the environment in which the 
photographing apparatus is disposed at a glance. Since the 
positional information of the pan tilter, the angle of view 
of the zoom lens, and the moving range of the pan tilter 
are added as information to the picture, the user can easily 
know the state of the photographing apparatus. 
[0148] 

In addition, when the user designates a desired 
object in the panorama operation area, he or she can easily 
capture it in the f ieldof viewof thepicturetobephotographed . 
Moreover, by designating an object in the operation area, 
the user can precisely adjust the position that cannot be 
designated in the panorama operation area. In comparison 
with the conventional method of which the user operates 
direction keys while observing a monitored picture (namely, 
a picture is photographed through a feed-back operation and 
an experience) , accordingtothepresent invention, adesired 
object can be displayed at the center of the operation area 
with the clicking operation of the mouse. 
[0149] 

In addition, according to the present invention, 
sincethepositiontowhichthepantiltermovescanbepredicted 
beforehand, on a communication line that causes picture and 
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informat ion data to be delayed and/ or lost (suchas Internet ) , 
the user can seamlessly operate the pan tilter camera . Thus , 
according to the present invention, the pan tilter camera 
can be easily operated with high visibility. 
[Brief Description of the Drawings] 
[Fig. 1] 

An external view for explaining a systemaccording 
to an embodiment of the present invention. 
[Fig. 2] 

A schematic diagram for explaining a screen of 
amoni tor according to the embodiment of thepresent invention . 
[Fig. 3] 

Ablock diagram showing the structure of the system 
according to the embodiment of the present invention. 
[Fig. 4] 

A schematic diagram for explaining a method for generating 
a panorama picture according to the embodiment of the present 
invention. 

[Fig. 5] 

A schematic diagram for explaining a method for 
generating a panorama picture according to the embodiment 
of the present invention. 
[Fig. 6] 

A schematic diagram for explaining a method for 
generating a panorama picture according to the embodiment 
of the present invention. 
[Fig. 7] 
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A schematic diagram for explaining a method for 
generating angular information of a pan tilter camera with 
positional coordinates inapanoramaoperationareaaccording 
to the embodiment of the present invention. 
5 [Fig. 8] 

A schematic diagram for explaining a plane - 
spherical surface converting method according to the 
embodiment of the present invention. 
[Fig. 9] 

10 A schematic diagram for explaining a coordinate 

converting method in the operation area according to the 
embodiment of the present invention. 
[Fig. 10] 

A schematic diagram for explaining a coordinate 
15 converting method in the panorama operation area according 

to the embodiment of the present invention. 
[Fig. 11] 

A schematic diagram for explaining positional 
information and angular information of a pan tilter camera 
20 according to the embodiment of the present invention. 

[Fig. 12] 

A schematic diagram for explaining angular 
coordinatesof thepantiltercameraandpos it ional coordinates 
in the panorama operation area according to the embodiment 
25 of the present invention; 

[Fig. 13] 

A schematic diagram for explaining the angle of 



view of the pan tilter camera and a frame in the panorama 
operation area according to the embodiment of the present 
invention. 

[Fig. 14] 

A graph for explaining a conversion method of zoom 
data and magnification data according to the embodiment of 
the present invention. 
[Fig. 15] 

A flow chart showing an example of the overall 
process according to the embodiment of the present invention . 
[Fig. 16] 

A flow chart showing an example of the process 
of a timer event according to the embodiment of the present 
invention. 

[Fig. 17] 

A flow chart showing an example of the process 
of a mouse moving event according to the embodiment of the 
present invention. 
[Fig. 18] 

A flow chart showing an example of the process 
of a mouse button down event according to the embodiment 
of the present invention. 
[Fig. 19] 

A flow chart showing another example of the process 
of a mouse button down event according to the embodiment 
of the present invention. 
[Fig. 20] 



A flow chart showing an example of the process 
of a mouse up/down event according to the embodiment of the 
present invention . 
[Fig. 21] 

A schematic diagram showing the structure of a 
system according to a second embodiment of the present 
invention. 

[Fig. 22] 

Ablock diagram showing the structure of the system 
according to the second embodiment of the present invention . 
[Fig. 23] 

A schematic diagram for explaining a controller 
for a photographing apparatus . 
[Description of Reference Numerals] 

1 . . . Computer, 2 . . . Monitor, 6A . . . Operation area, 6B . . . 
Panorama operation area, 7 . . . Cursor, 9 . . . Server, 11 . . . 
Camera portion, 12 . . . Pan tilter portion, 13 . . . TV monitor, 
14 . . . Pointing device, 15 . . . Lens block, 16 . . . Zoom lens, 
17 ... Zoom portion, 18, 26, 27 ... Motor, 19 ... Solid 
state imagepickupdevice, 20 . . . Signal separating/automatic 
gain adjusting circuit, 21 . . . A/D converter, 22 . . . Signal 
processing, 23 . . . Mode controller , 24 . . . Camera controller , 
25 . . . Pan tilter controller, 28 . . . Pan tilter, 29 . . . Video 
capture, 30 . . . Storing portion, 31 . . . Controlling portion 
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[Title of Document] Abstract 
[Abstract] 

[Subject] 

To display moving range of a pan tiliter camera 
as a panorama picture, and easily obtain a picture which 
is designated referring to the panorama picture from the 
pan tilter camera. 

[Solving Means] 

A picture photographed by a camera portion 11 is 
sent to a video capturing portion 29 of a computer 1. The 
picture is displayed in an operation area 6A of a monitor 
2 . Apanorama picture of which pictures in part or all moving 
range of a pan tiler 28 are combined is displayed in a panorama 
operation area 6B. A pan tilter portion 12 sends positional 
information of pan and tilt to the computer 1 through a mode 
controller 23. With a mouse 8, the operation area 6A and 
6B are operated so as to select an object. The computer 
1 obtains data for driving the pan tilter 28. Thus, the 
selected object is displayed at the center of the operation 
area . 

[Selected Drawing] Fig. 3 
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Translation of Drawings 



FIG. 1 

3 ... PAN TILTER CAMERA 
6E ... PANORAMA GENERATION 
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FIG. 2 

6 A . . . OPERATION AREA 

6E ... PANORAMA GENERATION 

6B . . . PANORAMA OPERATION AREA 



15 



20 



25 



FIG. 
(A) 
(B) 
(C) 
(D) 
(E) 
15 . 

22 . 
11 . 
13 . 
28 . 

26 . 

27 . 

24 . 

25 . 

23 . 



VIDEO SIGNAL 

DRIVE COMMAND 

POSITIONAL INFORMATION 

CONTROL, POSITIONAL INFORMATION 

ZOOM 
LENS BLOCK 
SIGNAL PROCESSING 
CAMERA PORTION 
TV MONITOR 
PAN TILTER 
PAN MOTOR 
TILT MOTOR 
CAMERA CONTROLLER 
PAN TILTER CONTROLLER 
MODE CONTROLLER 
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2 . . . MONITOR 

6A . . . OPERATION AREA 

6B . . . PANORAMA OPERATION AREA 

31 ... CONTROLLING PORTION 

2 9 ... VIDEO CAPTURE 

3 0 .. STORING PORTION 
1 . . . COMPUTER 

14 ... POINTING DEVICE 



10 FIG. 6 

(1) ... PICTURE CONVERTED INTO SPHERICAL SURFACE IS 
NORMALIZED WITH LATITUDE AND LONGITUDE 

(2) ... CAPTURED PICTURE 

(3) ... OBTAINED PICTURE 

15 

FIG. 7 

(1) ... PAN ANGLE 

(2) ... COORDINATE DATA 

(3) ... TILT ANGLE 

2 0 (4) . . . COORDINATE DATA 



FIG. 11 

(1) ... PAN ANGLE 

( 2 ) ... PAN DATA 
25 (3) . . . TILT ANGLE 

(4) ... TILT DATA 
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FIG. 12 

(1) ... PAN ANGLE 

(2) ... COORDINATE DATA 

(3) ... TILT ANGLE 

5 (4) . . . COORDINATE DATA 





FIG. 


13 




(1) 


. . . HORIZONTAL ANGLE OF VIEW (\|r) 




(2) 


. . . VERTICAL ANGLE OF VIEW (a) 


10 


(3) 


. . . PAN ANGLE 




(4) 


. . . COORDINATE DATA 




(5) 


. . . PAN ANGLE 




(6) 


. . . COORDINATE DATA 




(7) 


... HORIZONTAL ANGLE OF VIEW (s) 


15 


(8) 


. . . VERTICAL ANGLE OF VIEW (t) 



20 



25 



FIG. 14 

(1) ... INFORMATION OF LENS MAGNIFICATION 

(2) ... INTERNAL INFORMATION OF ZOOM LENS 

FIG. 15 
(A) ... START 



SI 
S2 
S3 
S4 
S5 



INITIALIZE OPERATION AREA ON MONITOR. 
SET TIMER EVENT. 

DETERMINE TYPE OF EVENT THAT OCCURS. 
TIMER EVENT 

MOUSE BUTTON DOWN EVENT 
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56 ... MOUSE BUTTON UP EVENT 

57 ... MOUSE MOVE EVENT 





FIG 


. 16 


5 


(A) 


. . . START 




(B) 


... NO 




(C) 


. . . YES 




(D) 


... NO 




(E) 


. . . YES 
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(F) 


. . . YES 




(G) 


... NO 




(H) 


. . . YES 




(I) 


... NO 




(J) 


. . . END 


15 


Sll 


. . . HAS COMMUNICATION PORT BEEN SET ? 




S12 


... IS DATA STORED IN RECEIVE BUFFER ? 




S13 


. . . ANALYZE RECEIVE DATA, OBTAIN PAN TILTER POSITION 




(p, t) AND ZOOM POSITION (z) , AND CALCULATE PAN TILTER ANGLE 




(9, 


q>) AND ZOOM MAGNIFICATION (y) . 


20 


S14 


... IS SEND REQUEST FLAG FlagSo TRUE ? 
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515 . . . SEND PAN TILER, ZOOM POSITION REQUEST COMMUNICATION 
DATA. 

518 ... SET COMMUNICATION PORT. 

519 . . . SEND DATA STORED IN SEND BUFFER AND RESET FlagSo 
TO FALSE. 

516 . . . HAS PAN TILTER DATA VARIED ? 

517 . . . HAS ZOOM DATA VARIED ? 
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S20 . . . REDRAW FRAME IN PANORAMA PICTURE AREA. 
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FIG. 17 

(A) ... START 

(B) ... NO 

(C) ... YES 

(D) ... NO 

(E) ... YES 

(F) ... END 

S21 ... IS MOUSE POINTER PRESENT IN OPERATION AREA ? 
S23 . . . OBTAIN COORDINATES (§, r\) OF POSITIONOFMOUSE POINTER 
IN OPERATION AREA. 

S25 . . . ISMOUSE POINTER PRESENT IN PANORAMAOPERATIONAREA ? 



15 FIG. 18 

(A) ... START 

(B) ... END 

532 ... OBTAIN (a, p) WITH EACH DATA. 

533 . . . CONVERT (a, p) INTO PAN TILTER POSITIONAL COORDINATES 
20 (PNew, TNew) , STORE THEM AS ABSOLUTE POSITION DRIVE REQUEST 

COMMUNICATION DATA TO COMMUNICATION BUFFER, AND SET FlagSo 
TO TRUE. 

S35 ... OBTAIN (a, p) WITH EACH DATA. 



25 FIG. 19 

(A) ... START 

(B) ... END 
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542 ... SET START POINT OBTAIN FLAG FlagRstart TO TRUE. 

543 ... OBTAIN COORDINATES (ml, nl) AT THE POINT. 

S45 ... SET START POINT OBTAIN FLAG FlagPstart TO TRUE. 



5 FIG. 20 

(A) ... START 

(B) ... END 

S53 ... OBTAIN COORDINATES (m2, n2 ) AND THEREBY (£, i\) AT 
THIS POINT. 

10 S54 ... OBTAIN (ot, (3) WITH EACH DATA. 

S55 . . . CONVERT (a, (J) INTO PAN TILTER POSITIONAL COORDINATES 
(PNew, TNew) , STORE THEM AS ABSOLUTE POSITION DRIVE REQUEST 
COMMUNICATION DATA TO COMMUNICATION BUFFER, AND SET FlagSo 
TO TRUE. 

15 S56 . . . SET START POINT OBTAIN FLAG FlagStart TO FALSE AND 

FlagPstart TO FALSE. 

559 ... OBTAIN COORDINATES (m2, n2) AND THEREBY r]) AT 
THIS POINT. 

560 ... OBTAIN (a, (3) WITH EACH DATA. 

20 

FIG. 21 

3 ... PAN TILTER CAMERA 
9 . . . SERVER 

6E ... PANORAMA GENERATION 

25 

FIG. 22 

(A) ... CAMERA CONTROL 
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. . . VIDEO SIGNAL 

. . . CONTROL, POSITIONAL INFORMATION 

. . . PICTURE DATA 

. . . GUI PANEL 

. . . STATE 

. . . COMMAND 

. . . COMMAND 

. . CAMERA PORTION 

. . SIGNAL PROCESSING 

. . TV MONITOR 

. . PAN TILTER PORTION 

. . MODE CONTROLLER 

. MONITOR 

. . OPERATION AREA 

. . PANORAMA OPERATION AREA 

. . PANORAMA GENERATION 

. COMPUTER 

. . CONTROL B 

. SERVER 

. . . CONTROL A 

. . . VIDEO CAPTURE 

. . . STORING 

. . POINTING DEVICE 

23 

. PAN TILTER CAMERA 
. SERVER 
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